Skip to content

feat(sql_execution): Implement retry logic for userpod API#89

Draft
tkislan wants to merge 5 commits intomainfrom
tk/userpod-api-retry-5xx
Draft

feat(sql_execution): Implement retry logic for userpod API#89
tkislan wants to merge 5 commits intomainfrom
tk/userpod-api-retry-5xx

Conversation

@tkislan
Copy link
Copy Markdown
Contributor

@tkislan tkislan commented Apr 2, 2026

  • Added a new function _create_retry_session to create a requests session with retry capabilities for handling 5xx errors on POST requests.
  • Updated _generate_temporary_credentials and _get_federated_auth_credentials to use the new retry session for making requests.
  • Introduced unit tests to verify the retry session configuration and its usage in the credential generation functions.

Summary by CodeRabbit

  • New Features

    • Added automatic retry for credential and federated-auth POST requests on transient server errors (500, 502, 503, 504) with backoff and limited retries to improve reliability.
  • Tests

    • Added tests covering retry configuration, retry/backoff behavior across success and failure scenarios, and validation that POST requests use the retry-enabled session with expected timeouts and headers.

- Added a new function `_create_retry_session` to create a requests session with retry capabilities for handling 5xx errors on POST requests.
- Updated `_generate_temporary_credentials` and `_get_federated_auth_credentials` to use the new retry session for making requests.
- Introduced unit tests to verify the retry session configuration and its usage in the credential generation functions.
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Apr 2, 2026

📝 Walkthrough

Walkthrough

Added _create_retry_session() returning a requests.Session configured with an HTTPAdapter and Retry (status_forcelist {500,502,503,504}, total=3, backoff_factor=0.5). _generate_temporary_credentials(integration_id) -> tuple[str, str] and _get_federated_auth_credentials(...) now use session = _create_retry_session() and call session.post(...) (preserving timeout=10, headers, response.raise_for_status(), and JSON parsing). Tests updated to mock _create_retry_session() and to exercise retry behavior via low-level connection/mocked sleep.

Sequence Diagram(s)

sequenceDiagram
    participant Caller as Caller
    participant Creator as _create_retry_session()
    participant Session as RetrySession
    participant Auth as Auth Server

    Caller->>Creator: request retry-enabled session
    Creator-->>Caller: configured requests.Session
    Caller->>Session: session.post(url, headers, timeout=10)
    Session->>Auth: HTTP POST (may retry on 500/502/503/504)
    Auth-->>Session: HTTP response(s)
    Session-->>Caller: response object
    Caller->>Caller: response.raise_for_status() and parse JSON
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Updates Docs ⚠️ Warning PR adds retry logic without documentation updates in docs/ directory or primary repos. Add documentation for retry mechanism, configuration, and affected API calls to docs/user/ or docs/dev/ and primary repositories.
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed Title clearly summarizes the main change: implementing retry logic for userpod API calls in sql_execution module.
Docstring Coverage ✅ Passed Docstring coverage is 91.67% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 2, 2026

📦 Python package built successfully!

  • Version: 2.2.0.dev8+30d806e
  • Wheel: deepnote_toolkit-2.2.0.dev8+30d806e-py3-none-any.whl
  • Install:
    pip install "deepnote-toolkit @ https://deepnote-staging-runtime-artifactory.s3.amazonaws.com/deepnote-toolkit-packages/2.2.0.dev8%2B30d806e/deepnote_toolkit-2.2.0.dev8%2B30d806e-py3-none-any.whl"

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@deepnote_toolkit/sql/sql_execution.py`:
- Around line 267-278: The cache lookup in _request_cache_info_from_webapp is
using requests.get directly and thus misses the retry behavior defined in
_create_retry_session; modify _create_retry_session (or its Retry config) to
include GET in allowed_methods (or use allowed_methods=None to cover all
idempotent methods) and then replace the direct requests.get call inside
_request_cache_info_from_webapp with a call through a session returned by
_create_retry_session (e.g., session.get(...)) so cache lookups get the same 5xx
retry protection as POSTs.
- Line 281: Update the signature of
_generate_temporary_credentials(integration_id) to include an explicit return
type annotation (e.g., -> Dict[str, Any]) and add the corresponding typing
imports (from typing import Dict, Any) at the top of the module; ensure the
annotated type matches the actual returned structure from
_generate_temporary_credentials so the function signature and implementation
remain consistent.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: bf6b45a7-61ec-4a87-a583-c7d418b8fe72

📥 Commits

Reviewing files that changed from the base of the PR and between 46807ba and 39baf42.

📒 Files selected for processing (2)
  • deepnote_toolkit/sql/sql_execution.py
  • tests/unit/test_sql_execution.py

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 2, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.45%. Comparing base (46807ba) to head (1f804b1).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #89      +/-   ##
==========================================
+ Coverage   74.32%   74.45%   +0.13%     
==========================================
  Files          94       94              
  Lines        5534     5543       +9     
  Branches      824      824              
==========================================
+ Hits         4113     4127      +14     
+ Misses       1155     1150       -5     
  Partials      266      266              
Flag Coverage Δ
combined 74.45% <100.00%> (+0.13%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@deepnote-bot
Copy link
Copy Markdown

deepnote-bot commented Apr 2, 2026

🚀 Review App Deployment Started

📝 Description 🌐 Link / Info
🌍 Review application ra-89
🔑 Sign-in URL Click to sign-in
📊 Application logs View logs
🔄 Actions Click to redeploy
🚀 ArgoCD deployment View deployment
Last deployed 2026-04-03 19:03:29 (UTC)
📜 Deployed commit 58aa085d540e3f1ce36c8e077d4bf8fbc5b20e24
🛠️ Toolkit version 30d806e

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/unit/test_sql_execution.py`:
- Around line 1055-1065: Convert the context-manager style mocking of
deepnote_toolkit.sql.sql_execution._create_retry_session (currently using with
mock.patch(...) as mock_create_session in the test block) to the decorator style
`@mock.patch` on the test function signature (e.g., add
`@mock.patch`("deepnote_toolkit.sql.sql_execution._create_retry_session") and a
mock_create_session parameter) and remove the inner with-block; apply the same
change to the similar block around lines 1089-1099 so both tests use
decorator-style mocking for consistency with
test_get_federated_auth_credentials_returns_validated_response.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 469a19f4-668a-4b55-8dfc-c8551fc61827

📥 Commits

Reviewing files that changed from the base of the PR and between 39baf42 and aa99cc3.

📒 Files selected for processing (1)
  • tests/unit/test_sql_execution.py

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

♻️ Duplicate comments (1)
deepnote_toolkit/sql/sql_execution.py (1)

280-280: ⚠️ Potential issue | 🟡 Minor

Add parameter type hint.

Return type added, but integration_id lacks type annotation per coding guidelines.

🔧 Fix
-def _generate_temporary_credentials(integration_id) -> tuple[str, str]:
+def _generate_temporary_credentials(integration_id: str) -> tuple[str, str]:
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@deepnote_toolkit/sql/sql_execution.py` at line 280, The function
_generate_temporary_credentials has a return type but the parameter
integration_id is missing a type annotation; update the function signature for
_generate_temporary_credentials to add the appropriate type (e.g.,
integration_id: str) to comply with the coding guidelines and any static type
checks, keeping the return annotation tuple[str, str] unchanged and adjusting
any callers only if they rely on a different type.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@deepnote_toolkit/sql/sql_execution.py`:
- Around line 270-274: The Retry instance named "retries" is missing
allowed_methods so POST requests won't be retried; update the Retry(...) call
(the retries variable in sql_execution.py) to include an allowed_methods
parameter that includes "POST" (for example
allowed_methods=frozenset({"GET","POST","PUT","DELETE","HEAD","OPTIONS","TRACE"})
or at minimum allowed_methods=frozenset({"POST"}) depending on desired scope) so
POST requests are retried per the backoff/status_forcelist settings.

In `@tests/unit/test_sql_execution.py`:
- Around line 1026-1040: The Retry configuration in _create_retry_session is
missing the allowed_methods parameter so POST retries aren't enabled; update the
Retry(...) construction inside the _create_retry_session function to include
allowed_methods=["DELETE","GET","HEAD","OPTIONS","POST","PUT","TRACE"] (or an
equivalent set that includes "POST"), then ensure that same Retry instance is
used when mounting the HTTP/HTTPS adapters so
adapter.max_retries.allowed_methods contains "POST".

---

Duplicate comments:
In `@deepnote_toolkit/sql/sql_execution.py`:
- Line 280: The function _generate_temporary_credentials has a return type but
the parameter integration_id is missing a type annotation; update the function
signature for _generate_temporary_credentials to add the appropriate type (e.g.,
integration_id: str) to comply with the coding guidelines and any static type
checks, keeping the return annotation tuple[str, str] unchanged and adjusting
any callers only if they rely on a different type.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: db1a3ece-0680-4ebe-a62e-d27e219521b2

📥 Commits

Reviewing files that changed from the base of the PR and between aa99cc3 and 01be651.

📒 Files selected for processing (2)
  • deepnote_toolkit/sql/sql_execution.py
  • tests/unit/test_sql_execution.py

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (1)
deepnote_toolkit/sql/sql_execution.py (1)

281-281: ⚠️ Potential issue | 🟡 Minor

Missing type hint for integration_id.

Per coding guidelines, explicit type hints are required for parameters.

Proposed fix
-def _generate_temporary_credentials(integration_id) -> tuple[str, str]:
+def _generate_temporary_credentials(integration_id: str) -> tuple[str, str]:
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@deepnote_toolkit/sql/sql_execution.py` at line 281, The function
_generate_temporary_credentials is missing a parameter type hint for
integration_id; update the signature to include an explicit type (e.g.,
integration_id: str) so it becomes def
_generate_temporary_credentials(integration_id: str) -> tuple[str, str]: and
adjust any callers or tests if they rely on a different type to keep annotations
consistent.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/unit/test_sql_execution.py`:
- Around line 1026-1092: Add a unit test for _create_retry_session that does not
mock it so you can inspect the session object's retry configuration; call
deepnote_toolkit.sql.sql_execution._create_retry_session(), get the returned
requests.Session, locate its mounted HTTPAdapter (e.g.,
session.get_adapter("https://")), and assert the adapter's max_retries
(requests.packages.urllib3.util.retry.Retry or similar) has total==3,
backoff_factor==0.5, status_forcelist contains the expected status codes, and
allowed_methods includes the expected idempotent methods.

---

Duplicate comments:
In `@deepnote_toolkit/sql/sql_execution.py`:
- Line 281: The function _generate_temporary_credentials is missing a parameter
type hint for integration_id; update the signature to include an explicit type
(e.g., integration_id: str) so it becomes def
_generate_temporary_credentials(integration_id: str) -> tuple[str, str]: and
adjust any callers or tests if they rely on a different type to keep annotations
consistent.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: e9bf096b-e2bf-4305-8062-6bbd7168cd64

📥 Commits

Reviewing files that changed from the base of the PR and between 01be651 and 790be77.

📒 Files selected for processing (2)
  • deepnote_toolkit/sql/sql_execution.py
  • tests/unit/test_sql_execution.py

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/unit/test_sql_execution.py`:
- Around line 1226-1261: The test
test_federated_auth_exhausts_retries_on_persistent_5xx declares mock_retry_sleep
but never asserts it; update this test to assert that mock_retry_sleep was
invoked the expected number of times (3 sleeps for 4 failed attempts) after
calling _get_federated_auth_credentials("test-id", "auth-context-token") and
after the RetryError assertion; reference the mock object name mock_retry_sleep
and the test function test_federated_auth_exhausts_retries_on_persistent_5xx
when adding mock_retry_sleep.assertEqual(mock_retry_sleep.call_count, 3) (or
equivalent assert_called_count).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 414980d6-b46e-4a59-bebc-7b7b56420428

📥 Commits

Reviewing files that changed from the base of the PR and between 790be77 and 1f804b1.

📒 Files selected for processing (1)
  • tests/unit/test_sql_execution.py

Comment on lines +1226 to +1261
@mock.patch("urllib3.util.retry.Retry.sleep", return_value=None)
@mock.patch("urllib3.connectionpool.HTTPConnectionPool._make_request")
@mock.patch("deepnote_toolkit.sql.sql_execution.get_project_auth_headers")
@mock.patch("deepnote_toolkit.sql.sql_execution.get_absolute_userpod_api_url")
def test_federated_auth_exhausts_retries_on_persistent_5xx(
self,
mock_get_url,
mock_get_headers,
mock_make_request,
mock_retry_sleep,
):
"""All 4 attempts return 504 - must raise ``RetryError``."""
import requests
from urllib3 import HTTPResponse as Urllib3Response

from deepnote_toolkit.sql.sql_execution import _get_federated_auth_credentials

mock_get_url.return_value = (
"https://api.example.com/integrations/federated-auth-token/test-id"
)
mock_get_headers.return_value = {"Authorization": "Bearer token"}

mock_make_request.side_effect = [
Urllib3Response(
body=io.BytesIO(b"Gateway Timeout"),
status=504,
headers={},
preload_content=False,
)
for _ in range(4)
]

with self.assertRaises(requests.exceptions.RetryError):
_get_federated_auth_credentials("test-id", "auth-context-token")

self.assertEqual(mock_make_request.call_count, 4)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Missing assertion on mock_retry_sleep.

mock_retry_sleep is declared but unused. The parallel test for _generate_temporary_credentials asserts mock_retry_sleep.call_count == 3. Add the same assertion here.

🔧 Proposed fix
         with self.assertRaises(requests.exceptions.RetryError):
             _get_federated_auth_credentials("test-id", "auth-context-token")

         self.assertEqual(mock_make_request.call_count, 4)
+        self.assertEqual(mock_retry_sleep.call_count, 3)
🧰 Tools
🪛 Ruff (0.15.9)

[warning] 1235-1235: Unused method argument: mock_retry_sleep

(ARG002)


[warning] 1258-1258: Use pytest.raises instead of unittest-style assertRaises

Replace assertRaises with pytest.raises

(PT027)


[warning] 1261-1261: Use a regular assert instead of unittest-style assertEqual

Replace assertEqual(...) with assert ...

(PT009)

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/unit/test_sql_execution.py` around lines 1226 - 1261, The test
test_federated_auth_exhausts_retries_on_persistent_5xx declares mock_retry_sleep
but never asserts it; update this test to assert that mock_retry_sleep was
invoked the expected number of times (3 sleeps for 4 failed attempts) after
calling _get_federated_auth_credentials("test-id", "auth-context-token") and
after the RetryError assertion; reference the mock object name mock_retry_sleep
and the test function test_federated_auth_exhausts_retries_on_persistent_5xx
when adding mock_retry_sleep.assertEqual(mock_retry_sleep.call_count, 3) (or
equivalent assert_called_count).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants